Recording dose data from drug injection devices using optical character recognition (ocr)

ABSTRACT

A method of recording a medicament dose using a data collection device comprises capturing, by a video camera of said data collection device, a video showing a medicament dose indicator of a medicament delivery device, adjusting a scale of an image of said medicament dose indicator in said video, adjusting said image for skew of one or more characters displayed on a component of the medicament delivery device in said video, determining the position of at least one of said one or more characters in the image, identifying the at least one character using optical character recognition and determining a medicament dose shown by the medicament dose indicator based on a result of said optical character recognition. The method may include determining whether more than one delivery of medicament is recorded in the video and, if so, whether said more than one delivery includes one or more prime shots, so that an overall dosage delivered to a user may be determined based on multiple determined medicament doses. A wearable electronic device comprising a video camera may be used to obtain and analyze the video, for example, using software provided in an “app”. The wearable electronic device may be configured to be worn on the head of a user, to capture the video from the user&#39;s point of view.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/520,053, filed on Apr. 18, 2017, which is a U.S. national stageapplication under 35 USC § 371 of International Application No.PCT/EP2015/073860, filed on Oct. 15, 2015, which claims priority toEuropean Patent Application No. 14189706.6 filed on Oct. 21, 2014, theentire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to data collection from a medicamentdelivery device. In particular, the present disclosure relates to amethod and a data collection system for recording a medicament dose.

BACKGROUND

A variety of diseases exists that require regular treatment by injectionof a medicament. Such injection can be performed by using injectiondevices, which are applied either by medical personnel or by patientsthemselves. As an example, type-1 and type-2 diabetes can be treated bypatients themselves by injection of insulin doses, for example once orseveral times per day. For instance, a pre-filled disposable insulin pencan be used as an injection device.

Alternatively, a re-usable pen may be used. A re-usable pen allowsreplacement of an empty medicament cartridge by a new one. Either penmay come with a set of one-way needles that are replaced before eachuse. The insulin dose to be injected can then for instance be manuallyselected at the insulin pen by turning a dosage knob and observing theactual dose from a dosage window or display of the insulin pen. The doseis then injected by inserting the needle into a suited skin portion andpressing an injection button of the insulin pen.

To be able to monitor insulin injection, for instance to prevent falsehandling of the insulin pen or to keep track of the doses alreadyapplied, it is desirable to measure information related to a conditionand/or use of the injection device, for example, one or more of theinjected insulin type, dose and timing of the injection, in a mannerthat is reliable and accurate.

Data collection techniques may also be used for purposes other thanmonitoring insulin injections. For example, data may be collected inorder to monitor injections of other medicaments, other medicalactivities, such as the taking of tablet medication by a patient orinfusions, or for non-medical purposes, such as the monitoring ofequipment and/or its operation in a home or industrial environment forsafety reasons.

SUMMARY

According to one aspect, there is provided a method of recording amedicament dose using a data collection device, the method includingcapturing, by a video camera of the data collection device, a videoshowing a medicament dose indicator of a medicament delivery device,adjusting a scale of an image in said video, adjusting said image forskew of one or more characters displayed on a component of themedicament delivery device in said video, determining the position of atleast one of said one or more characters in the image, identifying theat least one character using optical character recognition, anddetermining a medicament dose indicated by the medicament dose indicatorbased on a result of said optical character recognition.

The obtaining of medicament delivery information from a video recordingmay provide a reliable record and/or monitoring of the administration ofmedicament to a patient, since it does not rely on specific input fromthe patient and so may avoid user error and/or user failure to recollectdetails of an injection.

The data collection device may be a portable electronic device. Inparticular, the data collection apparatus may be a wearable electronicdevice. Where the data collection device is a device that is commonlyavailable, medicament delivery may be reliably recorded and monitoredwithout requiring specialized, or dedicated, data collection devices.Also, since the user can utilize a device that is familiar to them fordata collection, the user may begin to record dosages without having tofamiliarize themselves, and learn to use, a new device.

Where the data collection device is a wearable device that is worn onthe head of the user, the captured video may correspond approximately tothe point of view of a user, increasing the likelihood that themedicament dose indicator will be shown in the captured video.

The method may comprise determining whether more than one delivery ofmedicament is recorded in said video. In response to a determinationthat more than one delivery of medicament is recorded in said video, atleast a second medicament dose based on at least one characteridentified using optical character recognition on at least a secondimage of said medicament dose indicator in said video may be determined,and an overall medicament dose received by a user determined based onsaid determined medicament dose and said determined second medicamentdose. Alternatively, or additionally, the method may determine whethersaid more than one delivery of medicament includes one or more primeshots, so that any determined medicament doses associated with a primeshot can be disregarded.

The method may include obtaining color information from said image andidentifying a type of medicament to be dispensed based on said colorinformation. The obtaining of color information may include obtaining acolor balance measurement based on reference color information providedon the medicament delivery device.

The medicament delivery device may be an injector pen including amovable component for selecting said amount of medicament to bedispensed.

This aspect also provides a computer program comprisingcomputer-readable instructions that, when executed by a processor,causes one or more of the above methods to be performed.

Such a computer program may be provided in the form of an “app” for awearable electronic device or other portable electronic device.

This aspect further provides a data collection apparatus for recording amedicament dose, comprising a video camera and a processing arrangementconfigured to capture a video using said video camera, said videoincluding at least one image of a medicament dosage indicator of amedicament delivery device, adjust a scale of said image, adjust saidimage for skew of one or more characters displayed on a component of themedicament delivery device, determine the position of at least one ofsaid one or more characters in the image, identify the at least onecharacter using optical character recognition and determine a dosageindicated by said medicament dose indicator based on a result of saidoptical character recognition.

This aspect also provides a medicament delivery system including such adata collection apparatus and the medicament delivery device.

The data collection apparatus may comprise a wearable electronic devicethat includes said video camera. Such a wearable electronic device maybe configured to be worn on the head of a user, so that the capturedvideo corresponds to the user's point of view. For example, the wearableelectronic device may be provided in a form that can be worn by the userin a similar manner to a pair of glasses or sunglasses.

The apparatus may be configured so that the video camera and processingarrangement are included in the same device, such as a wearable deviceas discussed above. However, in some embodiments, the video camera maybe provided in a device that is configured to transmit the capturedvideo to a computer via a network, such as a wireless local areanetwork, a personal area network, a cellular communication network orthe Internet, where the computer comprises said processing arrangement.

The processing arrangement may be configured to determine whether morethan one delivery of medicament is recorded in said video. Theprocessing arrangement may be configured to respond to a determinationthat more than one delivery of medicament is recorded in said video bydetermining at least a second medicament dose based on at least onecharacter identified using optical character recognition on at least asecond image of said medicament dose indicator in said video anddetermining an overall dosage received by a user based on saiddetermined medicament dose and said determined second medicament dose.Alternatively, or additionally, the processing arrangement may beconfigured to [[to]] respond to such a determination by determiningwhether said more than one delivery of medicament includes one or moreprime shots, so that any determined medicament doses associates withsaid one or more prime shots can be disregarded.

The processing arrangement may be configured to identify a color of atleast one component of the medicament delivery device and to determine atype of said medicament based on said color. In some embodiments, theprocessing arrangement may be configured to obtain a color balancemeasurement based on an image showing reference color informationprovided on the medicament delivery device.

BRIEF DESCRIPTION OF THE FIGURES

Example embodiments of the subject matter described herein will now bedescribed with reference to the accompanying figures, of which:

FIG. 1a shows an exploded view of a medicament delivery device;

FIG. 1b shows a perspective view of a portion of the medicament deliverydevice of FIG. 1 a;

FIG. 2 is a block diagram of a data collection device according to anembodiment;

FIG. 3 is a flowchart of a data collection method according to anembodiment;

FIGS. 4 and 5 each show a portion of a dosage window of the drugdelivery device of FIG. 1a , with examples of digits that may bedisplayed;

FIG. 6 shows an example of a binarized digits corresponding to digitsdisplayed in the dosage window;

FIGS. 7 and 8 show diagrammatically further examples of digits that maybe displayed in the dosage window; and

FIG. 9 shows a further example of digits that may be displayed in thedosage window where a primary digit row contains two digits; and

FIG. 10 is a flowchart of a data collection method according to anotherembodiment.

DETAILED DESCRIPTION

In the following, embodiments will be described with reference to aninsulin injection device. The present disclosure is however not limitedto such application and, as noted herein above, may equally well bedeployed with injection devices that eject other medicaments, or withother types of medicament delivery devices.

FIG. 1a is an exploded view of an injection device 1 which, in thisparticular example, represents Sanofi's Solostar® insulin injection pen.

The injection device 1 of FIG. 1a is a pre-filled, disposable injectionpen that comprises a housing 10 and contains an insulin container 14, towhich a needle 15 can be affixed. The needle is protected by an innerneedle cap 16 and an outer needle cap 17, which in turn can be coveredby a cap 18. An insulin dose to be ejected from injection device 1 canbe selected by turning the dosage knob 12, and the selected dose is thendisplayed via dosage window 13, for instance in multiples of so-calledInternational Units (IU), wherein one IU is the biological equivalent ofabout 45.5 micrograms of pure crystalline insulin (1/22 mg). An exampleof a selected dose displayed in dosage window 13 may for instance be 30IUs, as shown in FIG. 1a . It should be noted that the selected dose mayequally well be displayed differently.

The dosage window 13 may be in the form of an aperture in the housing10, which permits a user to view a limited portion of a number sleeve 70that is configured to move when the dosage knob 12 is turned. In orderto facilitate taking images of the numbers displayed in the dosagewindow 13, the number sleeve 70 may have a matte surface.

A label 19 is provided on the housing 10. The label 19 includesinformation about the medicament included within the injection device,including information identifying the medicament. The informationidentifying the medicament may be in the form of text. The informationidentifying the medicament may also be in the form of a color. Forexample, the label 19 may have a background, or include a shaded elementsuch as a border having a color that corresponds to a particular type ofmedicament that is provided in the injection device.

Alternatively, or additionally, the label may include a RFID tag orsimilar device that stores such information.

One or more parts of the injection device, such as an injection button11 or the dosage knob 12, may be formed of a material having a colorthat corresponds to the medicament. Optionally, a part of an insulincontainer (not shown) within the injection device 1 may include acolor-coded portion that indicates a medicament type and may be viewablethrough the dosage window 13.

The information identifying the medicament may additionally, oralternatively, be encoded into a barcode, QR code or the like. Theinformation identifying the medicament may also be in the form of ablack and white pattern, a color pattern or shading.

Turning the dosage knob 12 causes a mechanical click sound to provideacoustic feedback to a user. The numbered sleeve 70 mechanicallyinteracts with a piston in insulin container 14.

When needle 15 is stuck into a skin portion of a patient, and theninjection button 11 is pushed, the insulin dose displayed in the dosagewindow 13 will be ejected from injection device 1.

When the needle 15 of injection device 1 remains for a certain time inthe skin portion after the injection button 11 is pushed, a highpercentage of the dose is actually injected into the patient's body.Ejection of the insulin dose also causes a mechanical click sound, whichis however different from the sounds produced when using dosage knob 12.

Injection device 1 may be used for several injection processes untileither insulin container 14 is empty or the expiration date of injectiondevice 1 (e.g. 28 days after the first use) is reached.

Furthermore, before using injection device 1 for the first time, it maybe necessary to perform a so-called “prime shot” to remove air frominsulin container 14 and needle 15, for instance by selecting 2 IU ofinsulin and pressing injection button 11 while holding injection device1 with the needle 15 upwards. For simplicity of presentation, in thefollowing, it will be exemplarily assumed that the ejected dosessubstantially correspond to the injected doses, so that, for instancewhen making a proposal for a dose to be injected next, this dose equalsthe dose that has to ejected by the injection device. Nevertheless,differences (e.g. losses) between the ejected doses and the injecteddoses may of course be taken into account, particularly with regard to a“prime shot”.

FIG. 1b is a close-up of the end of the injection device 1. In theparticular example shown in FIG. 1, a locating rib 71 is located betweenthe viewing window 13 and the dosage knob 12.

FIG. 2 is a block diagram of a data collection device 20 according to anembodiment, that may be used to collect data, such as insulin type,dosage and timing of injection, from the injection device 1 of FIG. 1.

The data collection device 20 is an electronic device, equipped with abuilt-in video camera 21, and a processing arrangement 22 including oneor more processors, such as a microprocessor, a Digital Signal Processor(DSP), Application Specific Integrated Circuit (ASIC), FieldProgrammable Gate Array (FPGA) or the like. In this particular example,the data collection device 20 is a wearable electronic device, such as avideo camera device that may be worn by the user to record images fromtheir point of view, or a computing device that is mounted in a pair ofglasses or otherwise worn on the head or body of the user.

The data collection device 20 also includes memory units 23, 24,including a read-only memory 23 and a random access memory 24, which canstore software for execution by the processing arrangement 22. The datacollection device 20 also includes communications equipment 25, 26, suchas an antenna 25 and a transceiver 26, to permit bi-directionalcommunication with one or more of a cellphone network, a personal areanetwork, a local wireless network or WLAN, and the Internet. The datacollection device 20 further includes an input arrangement 27, 28, suchas a keypad 27 and/or a microphone 28, and an output arrangement 29, 30,such as a speaker 29 and/or a display 30. In some embodiments, the inputarrangement may include provide a keypad 27 in the form of atouch-sensitive element, or “touchpad” or as part of a touch-screen thatutilizes some or all of the display 31. The input arrangement mayalternatively, or additionally, include a motion sensor arrangement 31,such as one or more accelerometers, for detecting movement of the datacollection device 20. The data collection 20 also includes acommunications bus 32 allowing for communication between the videocamera 21, processing arrangement 22, memory units 23, 24,communications equipment 25, 26, input arrangement 27, 28, 31 and outputarrangement 29, 30.

The software stored in the memory units 23, 24 of the data collectiondevice 20 includes computer-readable instructions that, when executed bythe processing arrangement 22, causes the data collection device 20 torecord a video, and to process the video to obtain data regarding thetype of medicament in the injection device 1, a dose delivered by theinjection device 1 and, optionally, a time of delivery of themedicament. The software may be provided in the form of a softwareapplication, or “app”, that may be downloaded from a library or storeover the Internet.

An example method according to an embodiment will now be described withreference to FIGS. 3 to 9.

Starting at FIG. 3, step 3.0, the video camera 21 is controlled to beginrecording images of at least part of the injection device 1 (step s3.1),including the dosage window 13. In some embodiments, the app isconfigured to cause the processing arrangement 22 to provide guidance tothe user while the video is being recorded, by providing instructionsregarding positioning of the injection device 1 relative to the camera21 based on a “live view” of the field of view of the camera 21. Forexample, the app may be configured to guide the user to position thecamera 21 at a distance at which the camera 21 can obtain a good focuson the dosage window 13. In certain embodiments, such focus may beachieved when the distance between the camera and the injection device 1is approximately 30 cm.

Further, if medicament information is included only in another part ofthe injection device 1, such as the label 19, then the app may cause theprocessing arrangement 22 to instruct a user of the data collectiondevice 20, 33 to include that part of the injection device 1 byproviding a message on the display 29 and/or an announcement over thespeaker 30.

In this embodiment, the processing arrangement 22 then performspre-processing (step s3.2), to assess and, if required, improve videodata quality by executing the following steps:

-   -   Defective and bad pixel correction    -   Light correction    -   Distortion    -   Jitter

For example, an exposure control algorithm may adjust the operation ofthe video camera 21 to correct for images that are too bright or toodark by controlling exposure parameters for the video camera 21 and/orcontrolling additional lighting, where provided, for example, in anembodiment where the data collection device 20 includes a flash unit(not shown). It is noted that the pre-processing is an optional feature.The app may be designed to perform to the required standard withoutpre-processing of the image.

The distance between the injection device 1 and the video camera 21 isnot fixed and the orientation of the dosage window 13 and, whereprovided, the label 19, relative to the camera 21 may also vary. In viewof this, the processing arrangement 22 adjusts the scale of imageswithin the video recording so that the size of the characters displayedwithin the dosage window 13 are within a predetermined range (steps3.3).

The processing arrangement 22 may further adjust the images within thevideo recording by correcting skew of the characters displayed in thedosage window 13 based on the orientation of the injection device 1relative to the camera and/or any slanting of the characters displayedin the dosage window 13 (step s3.4). For instance, the numbers in thedosage window 13 might be slanted for ease of recognition andpositioning by a user, but may be easier to decode by the datacollection device 20 if the slant is removed.

The scale and skew adjustments may be based on an analysis of featuresof the injection device 1 of predetermined shape and/or size. Forexample, the processing arrangement 22 may identify the dosage window13, label 19, a logo (not shown) on the injection device 1 and/or otherfeatures of the injection device 1 in the image and, based oninformation regarding the expected shape and/or size of those features,adjust the scale and alignment of the images taken by the video camera21 and correct for skew of text and numbers included in the images.

In some embodiments, the app may also control the processing arrangement22 to determine whether the dosage window 13 and a particular part ofthe injection device 1, such as the label 19, that includes medicamentinformation are included in the video images (step s3.5). Thedetermination may be based on the scale of the injection device 1 in theimage and the location of one or more parts of the injection device 1within the image. The app may optionally provide guidance to the user toadjust the positioning of the injection device 1 relative to the videocamera 21 so that the relevant parts of the injection device 1 arewithin a field of view of the video camera 21.

The processing arrangement 22 then attempts to recognize characters fromat least one image of the dosage window from the video recording, usingOptical Character Recognition (OCR) software included in the app (steps3.6), in order to determine a number or other dosage indicationdisplayed in the dosage window 13 (step s3.7).

FIGS. 4 and 5 depict a portion of the dosage window 13, showing examplesof digits that may be displayed. In FIG. 4, a dosage has been dialedinto the injection device 1 such that a digit 34, in this case thenumber 6 indicating 6 IU, is displayed centrally in the dose window 13.In FIG. 5, a dosage of 7 IU has been dialed into the injection device 1such that digits 35, 36, representing the numbers 6 and 8 respectively,are both displayed in the dose window 13 and the space between thesenumbers occupies the central region of the dose window 13. In thisparticular embodiment, the processing arrangement 22 is configured toexecute an algorithm allowing both of the situations depicted in FIGS. 4and 5 to be decoded accurately.

The OCR process comprises the steps of:

-   -   Binarization    -   Segmentation    -   Pattern matching    -   Position calculation

Due to the high reliability requirements of the sensor device 2, theremay in some embodiments be two OCR algorithms that are operated inparallel. The two OCR algorithms have the same input (image) and areintended to provide the same output. They both perform similar stepshowever the individual methods used in each step may vary. These two OCRalgorithms may differ in one of the binarization, segmentation, patternmatching and position calculation steps or in more than one of thesesteps. Having two OCR-parts which use different methods to provide thesame result increases the reliability of the entire algorithm as thedata has been processed in two independent ways.

In the OCR process, the color or greyscale image obtained from the videocamera 21 and adjusted as described above is converted into a purelyblack and white image 37, such as that depicted in FIG. 6, through abinarization process. In an example where dark numbers are presented ona bright background in the dosage window, the black and white imagewould indicate the presence of digits 38, 39 with black pixels and theabsence of digits with white pixels, as shown in the example of FIG. 6.In some embodiments a fixed threshold is used to separate between blackand white pixels. Pixels that have a value at or above the thresholdbecome white, pixels below the threshold become black in the binarizedpicture. A high threshold will lead to artefacts (black parts in whiteareas), whereas a low threshold has the risk that in some cases parts ofdigits are missing. In some embodiments, the threshold is chosen so thatin no case are parts of digits are missing because the algorithm is ingeneral robust against artefacts (i.e. an accurate OCR process can beperformed in the presence of some artefacts). In tests where an imagewas analyzed using 256 grey values, a threshold value of 127 showed goodresults.

The use of a fixed threshold is possible where light correction has beenperformed, for example, in the pre-processing. The combination of thelight correction and the fixed threshold is similar to a windowed meanbinarization. A windowed mean binarization compares the pixel-value withthe mean value of the pixels of the area where it is located. Performingthe light correction step before the distortion and slant correctionsteps means that more information is available to be used for the OCRprocess, which has been shown to yield better results on the edges andcorners of the picture.

Alternatively, the Otsu threshold method may be applied to the capturedgreyscale image to produce a binary image similar to the black and whiteimage 37 shown in FIG. 6. In some alternative embodiments, thebinarization may be omitted and the OCR part of the algorithm may beperformed on the captured color or greyscale image.

Segmentation is then performed. The goal of this part of the algorithmis to determine the exact location of each visible or partly visiblenumber in the image. To achieve this, the algorithm defines theboundaries of the visible digits by finding the edges of the digits.This is generally accomplished in two steps, which may be performed inany order. Referring again to FIGS. 4 and 5, the processing arrangement22 may perform a “vertical projection” in which the pixel columns makingup the binarized image 37 are analyzed. Each pixel column is analyzedindividually and the sum of the number of black pixels in each column iscomputed. In some embodiments, only a pixel column having zero blackpixels defines the edge of a number.

Alternatively, a low threshold for the number of black pixels may be setto account for dirt, scratches and other disturbances. Difference valuesfor adjacent columns are calculated and the boundary having the greatestdifference represents the edge of the number. Additionally, the pixelcontent of overlapping groups of columns (e.g. three adjacent columns)may be calculated to aid in determining the horizontal edges of thenumbers.

The processing arrangement then performs a “horizontal projection” inwhich the pixel rows making up the binarized image 37 are analyzed. Thisproceeds in a similar manner to that as described above with regard tothe vertical projection.

The expected result of the horizontal projection is added to that of thevertical projection such that the edges of the visible numbers areidentified. The processing arrangement 22 may be pre-programmed with theexpected height (in pixel rows) of a full number, and so is able torecognize the presence of partially visible numbers.

In another embodiment, the “horizontal projection” and the “verticalprojection” may be based on an analysis where the sum of white pixels iscomputed, provided that the expected number of white pixels in each rowand column is known.

Knowing the exact location allows for using only the part of the imagewhich represents the visible number or numbers for the next steps in theOCR process. By this any impact of other objects besides the number,e.g. dirt, scratches and other disturbances, can be reduced.

Further, the total number of pixels to be processed in subsequent steps,e.g. in the pattern matching step, is also reduced. This helps reducingresource requirements. This also helps increasing performance. Inaddition, knowing the exact location also supports determining thevertical position relative to the center of the image.

The next step in the OCR process is to select one of the visible numbersto be decoded and identified. This is done by designating one of thenumbers as the “primary digit row”. The primary digit row is selectedbased on which visible number has the greatest height. This is becauseall of the numbers printed on the sleeve 70 have approximately the sameheight and it can be assumed that the number having the greatest heightwill be fully visible and therefore easy to decode with a high degree ofcertainty. In the example shown in FIG. 7, the number “6” has a greaterheight than the partially visible numbers above and below and so isselected as the primary digit row. In the example shown in FIG. 8, boththe numbers “6” and “8” are fully visible and have the same height. Inthis case, the uppermost number is selected as the primary digit row.The primary digit row is the number which is subsequently used todetermine the dose dialed into the injection device 1.

A standard injection device 1 for self administration of insulin caninject any number of units of medicament from 1 to 80 IU. Therefore, inorder to property decode the number identified as the primary digit row,it must be determined whether the number consists of one or two digits.

The processing arrangement 22 therefore performs a series of steps inorder to determine whether each number consists of one or two digits,and in the latter case, to separate the digits from each other. Theprocessing arrangement 22 may use the column pixel informationpreviously calculated for this purpose. In the example shown in FIG. 9,9 IU of medicament have been dialed into the injection device 1. Theexpected results of the horizontal and vertical projections are shown.The number “10” is of greater height than the number “8” and istherefore selected as the primary digit row.

After this the processing arrangement 22 determines whether the selectedprimary digit row is wider than a pre-defined “maximum digit width”value. The processing arrangement 22 may be pre-programmed withinformation relating to the expected size of the numbers in the capturedimages, so that a maximum expected width for a single digit can bedefined. In order to increase reliability, the maximum width may be setas a small number of pixel columns more than the widest number. If thewidth of the primary digit row is the maximum digit width or less, it isassumed that the row contains a single digit. If the primary digit rowis too wide to be a single digit, then a second vertical projection isthen performed on the primary digit row (rather than on the wholeimage). In addition, the expected width of each individual digit may beused to predict the point at which the separation should occur.

The exemplary field of view 900 shown in FIG. 9 is a diagrammaticrepresentation in which the numbers are well spaced. In otherarrangements, the numbers may be displayed quite close together in thedosage window, owing to limited available space and the need for thenumbers to be readable to a user. Thus, after binarization, the twodigits making up the number may not be cleanly separated, i.e. there maynot be a column having no black pixels between the two digits. This isthe case in the exemplary binarized image shown in FIG. 6, in which the“7” and “4” of the upper digits 37 do not have a pixel column betweenthem containing no black pixels. In this case, the expected width ofeach individual digit is again used to predict the point at which theseparation should occur. If the predicted column contains black pixels,then the deviations of this column from adjacent columns are calculatedto determine the best separation point. In this situation, as it is notclear whether the black pixels in the chosen separating column belong tothe left or right digit, they are ignored. This has been shown to have aminimal effect on the reliability of the OCR process to correctlyidentify the digits.

A pattern matching process is then performed to identify the digits inthe primary digit row. Templates for each number may be pre-programmedvia the app and the identified digits may then be compared to thesetemplates. In a straight forward approach the pattern matching could beperformed on a pixel-by-pixel basis. However, this may require highcomputing power and may be prone to position variation between the imageand the template. Where templates are used, the app may be configured tocause the processing arrangement 22 to perform other types ofmanipulation on the images numbers, for example by changing the size ofone or more digits, cropping the numbers to a defined pixel area andshearing numbers printed in a italic font into an upright position.These manipulations may be performed before a pattern matchingcomparison with the stored templates. Alternatively, these manipulationsmay be performed in preprocessing before the binarization process.Additional shading, distortion and exposure correction may also beperformed.

In some other embodiments, a feature recognition process is performed.Features may be horizontal, vertical or diagonal lines, curves, circlesor closed loops etc. Such features may be recognized in the image of theselected number and compared with templates.

In yet further embodiments, the pattern matching algorithm may be basedon a vector comparison process. For example, the templates may be in theform of vectors describing the position and length of each line(continuous run) of black pixels. In one example, the position andlength relate to the absolute position in the respective line. Inanother example, the position and length relate to a vertical lineextending through the center of the template. The captured binary imageof each digit may similarly be converted into vectors and compared witheach stored template in turn to find the best match. When comparing thevectors of the captured image with a particular digit template, anydeviations result in a penalty being applied for the likelihood of amatch between the image and that template. The magnitude of the penaltymay depend on the number of missing or extra black pixels in the imagecompared to the template.

After the digit image has been compared with each template and all ofthe penalties have been applied a decision is made as to which digit ispresent. In good optical conditions, the correct template will have avery low penalty, while all other templates will have a high penalty. Ifthe primary digit row consists of two digits, this process is performedon both digits and the processing arrangement 22 can then combine theoutcomes to produce a final result for the number.

Special measures may exist for certain digits. For example, “1” deviatessubstantially in width from all other digits resulting in commonmisdetections. To counter this, if a binary image of a digit is widerthan the expected width of “1”, then it receives an additional detectionpenalty when being compared with the stored vector template of “1”.

In some exceptional cases, if the confidence level in the result of thepattern matching of the primary digit row is below a certain threshold(e.g. 99%), then the processor may perform a second pattern matchingprocess on one or more of the other visible or partially visiblenumbers. Since the order of the numbers is known, this second patternmatching can act as a check that the first pattern matching returned thecorrect result.

If the confidence level in the result is still not high enough, thensteps s3.1 to s3.6 may be repeated for a second image from the videorecorded by the video camera 21. Alternatively, an error message may bedisplayed and, optionally, an instruction issued to the user through theoutput arrangement 29, 30 to continue capturing video images of thedosage window 13 or to enter dosage information manually, via the inputarrangement 27, 28.

If the digit or digits of the primary digit row have been successfullyidentified, a weighting function is applied in order to determine a dosedisplayed in the dosage window 13 (step s3.7). To formulate theweighting function, the vertical position of the primary digit rowrelative to the center of the dosage window 13 may be determined. Thismay be done by calculating the offset of the middle pixel row comprisingthe primary digit row relative to a pixel row representing a center lineof the dosage window 13 in the image.

For example, in some embodiments the optical sensor comprises arectangular 64×48 array of photosensitive elements. The resulting binaryimage is a pixel array having these same dimensions. The 24^(th) and/or25^(th) pixel row may be designated as the central row of the image. Theposition of the middle pixel row comprising the primary digit row isdetermined. The offset, in pixel rows, between the middle pixel rowcomprising the primary digit row and the central row or rows of theimage is then calculated. This offset may be positive or negativedepending on the direction of the offset. The offset is converted into afraction by dividing it by the distance (in pixel rows) betweensuccessive numbers before being applied to the determined numbersaccordingly. The offset therefore allows for determining the rotationalposition of the number relative to the sensor. If the central pixel rowof the primary digit row is the same as the central pixel row of theimage, then the offset is zero and the position is equal to the primarydigit row number. However, there is likely to be some offset in mostcircumstances.

The distance between successive numbers printed on the number sleeve 70is constant, since the numbers represent a dose which is related to adiscrete mechanical movement of the injection device mechanism.Therefore, the distance (in pixel rows) between successive numbers inthe captured image should also be constant. The expected height of thenumbers and spaces between the numbers may be pre-programmed into theapp. As an example, the expected height of each numbers may be 22 pixelsand the expected height of the spaces between the numbers may be 6pixels. Therefore, the distance between the central pixel rows ofsuccessive numbers would be 28 pixels.

Continuing this example, if the pixel rows are numbered sequentiallyfrom the top to the bottom of the image, the application of theweighting function may be defined mathematically as:

Position=primary digit row number+[2×offset/(expected height ofnumber+expected height of space)]

Where offset=image row number corresponding to the center of the dosagewindow−primary digit row central row number

Thus, if the primary digit row is in the upper half of the image, thenthe offset is positive and if the primary digit row is in the lower halfof the image, then the offset is negative. For example, if the numbershown in the primary digit row is “6” and the offset is zero, then thecalculated position would be:

Position=6+[2×0/(28)]=6

Thus a result of “6” would be returned as expected.

In another example, where 75 IU are dialed into the injection device 1,if the top number, “74”, is selected as the primary digit row and thereis a positive offset of 11 pixel rows according to the equation above,and again assuming a combined number/space height of 28 pixels, thecalculated position would be:

Position=74+[2×11/(28)]=74.79

This result is then rounded up to the nearest whole number, to give aposition determination of “75” as expected.

The skilled person will appreciate that the above described weightingfunction and position determination represents only one example and thatnumerous other calculation methods may be used to arrive at the sameresult. The skilled person would also appreciate that the abovedescribed mathematical calculation may be modified and improved toreduce the computation time. Thus the exact form of the weightingfunction is not essential to a definition of the present disclosure.

In some injection devices, due to space restrictions and the need forthe numbers to be of a certain size, only even numbers are presented inthe dosage window 13. In some other injection devices, only odd numbersmay be displayed. However, any number of units of medicament can bedialed into the injection device 1. In other injection devices, botheven and odd numbers may be presented and it may be possible to dialhalf-unit doses into the injection device. The injection device may belimited to a maximum dialed dose of 80 IU. Alternatively, only every3^(rd), 4^(th) or 5^(th) number may be displayed and doses between thenumbers may be indicated by tick marks. In view of this, the app mayinclude instructions for controlling the processing arrangement 22 toidentify the numbering sequence used in the injection device 1. Forexample, the user may be prompted to enter information regarding theinjection device 1 via a keypad 27 or touch-screen 33 or informationobtained from the image, for example from the text or a barcode on thelabel 19 may be used. The app may include a look-up table or otherinformation indicating the numbering sequences used for variousinjection devices 1. The processing arrangement 22 may then determinethe selected dose based on both OCR data and the appropriate numberingsequence for the injection device 1. Alternatively, or additionally, amodified form of the weighting function may be used, as the height ofthe numbers and size of the space between the numbers may also bemodified.

The method may optionally include post-processing, such as performingsanity checks and hysteresis calculations. Alternatively, the result ofthe OCR process may be finalized without post-processing.

The OCR process, as described above in steps s3.6 to s3.7, is thenrepeated on at least one subsequent image from the video, in order todetermine whether the dialed dosage has changed (step s3.8). This stepis intended to determine whether the dosage identified in step s3.7represents a dosage to be delivered by the injection device 1 or whetherthe user has continued to adjust the dialed dosage using the knob 12. Insome embodiments, the processing arrangement 22 is configured toidentify the last determined dosage in the video to be the actual dosagedelivered to the user. In other example embodiments, the processingarrangement 22 may identify an actual dosage delivered by the injectiondevice 1 based on the detection of the same dialed dosage in apredetermined number of successive, or a sequence of substantiallysuccessive, video frames in steps s3.6 to s3.7.

Information regarding the type of medicament may be obtained by one ormore of using OCR from a part of the image including the label 19,extracting and interpreting a barcode from a part of the image includingthe label 19 or by identifying a color of a part of the injection device1 that indicates the medicament type from the image (step s3.9).

In embodiments where color detection is used, the numbered sleeve 70 ofthe injection device 1 may be configured to provide reference colorinformation to allow one or more colors in the image to be identifiedcorrectly. For example, the background of the printed numbers may beused to provide a white balance level for calibrating the colors of thecomponents of the injection device 1 as shown in the image.

Information regarding the time at which the medicament was delivered bythe injection device 1 is obtained from timestamp data associated withthe video recording (step s3.10).

The information obtained from the video may then be stored in the randomaccess memory 24 of the data collection device 20 (step s3.11) and/ortransmitted to another device via a network, such as a cellphonenetwork, personal area network, WLAN or the Internet, using thecommunications equipment 25, 26 (step 3.12). On this manner, theadministration of medicament to a patient may be recorded and/ormonitored. The process ends at step s3.13.

Another embodiment of a method that may be used to collect from theinjection device 1 using the data collection device is shown in FIG. 10.

Starting at step 10.0, steps 10.1 to 10.10 correspond to steps 3.1 to3.10 respectively.

The method shown in FIG. 10 includes additional steps of analyzing thevideo recording to determine whether the injection device 1 is activatedto deliver more than one dosage of medicament. As noted above, theinjection device 1 may have an upper limit to the amount of medicamentthat may be delivered in one injection, such as 80 IU. In practice, morethan one actual injection may be needed in order for a user to receivetheir required dose of medicament. If the user requires a dose above amaximum dosage of the injection device 1, for instance, above 80 IU inthe example noted above, two injections might be administered in quicksuccession.

Alternatively, or additionally, the video may include images of a userperforming one or more “prime shots” to eliminate air from the injectorpen that should be disregarded when determining the dosage delivered tothe user.

At step s10.11, the processing arrangement 22 determines whether thevideo captured by the video camera 21 includes images of a furtherdosage being_dialed into the injector pen 1 by repeating the OCR processdescribed in relation to steps s10.6 and s10.7 of on one or moresubsequent images from the video. For example, when the SoloSTAR® pen,mentioned above, is used to administer an injection, the numberdisplayed in the dosage window 13 decreases to zero as the medicament isdelivered. Therefore, a sequence of images in the video showing achanging number in the dosage window 13 after a dialed dosage has beenidentified in step s10.7, or the subsequent detection of a non-zerocharacter in the dosage window 13, may be considered an indication thata further delivery of medicament is being performed by the user.

If a further delivery of medicament is indicated, the processingarrangement 22 then determines whether the previous delivery ofmedicament corresponded to a “prime shot” or an actual injection ofmedicament (step s10.12). A “prime shot” may be detected based on thedetermined dosage. For example, the detection of a dialed dosage of 2 IUor less in step s10.8 or s10.9 may be considered to correspond to a“prime shot”, rather than an injection received by the user.Alternatively, or additionally, the time stamp obtained at step s10.10and a time stamp associated with the image on which the detection of anew dosage being dialed into the injection device 1 is based in steps10.11, or the time between detections of further dosages in steps10.11, may be used to indicate whether a “prime shot” has beenperformed. The administration of an injection into a user may requirethe needle to remain in the user's body for several seconds, forexample, 10 seconds in certain insulin injections. Furthermore, where asecond injection is to be performed, the user would be expected tochange the needle 15 after the first injection, increasing the timeinterval between detections of dialed dosages by the OCR process insteps s10.6 to s10.7 and step s10.11. Where that time interval exceeds apredetermined limit, such as 10 seconds, a second injection may beindicated. If the time interval is less than the predetermined limit,then the processing arrangement 22 may determine that the previousactivation of the injection device 1 was a “prime shot”.

In embodiments where the OCR process is performed while the video isbeing captured by the video camera 21, the data collection device 20 mayoutput a request to the user to indicate whether a second injection isbeing made and/or whether a “prime shot” has been performed, via thedisplay 29 and/or the speaker 30. The user may respond to the requestvia the input arrangement 27, 28, 31.

If it is determined that only one injection was delivered at steps10.11, then the overall dosage received by the user is determined tocorrespond with the dosage detected in step s10.7.

If it is determined that a second injection has been delivered (steps10.11) and that neither injection was a ‘prime shot”, then the dosagesis determined in steps 10.7 and 10.11 are added together to determine anoverall dosage received by the user (step s10.13). However, if it isdetermined that one or more dosages detected in steps s10.7 and 10.11related to a “prime shot” (step s10.12), then the respective determineddosages is not included in the overall determined dosage (step s10.14)and a check for further injections is made (step s10.11).

In an alternative embodiment, steps 10.12 to s10.14 may be omitted and,where further dosages are detected (step s10.11), data relating todetected dosage may be stored and/or transmitted for later analysis.

Information regarding the type of medicament and time stamp informationmay be stored and/or transmitted (steps s10.15, 10.16) as describedabove in relation to steps 3.11 to 3.12 of FIG. 3. The process then ends(step s10.17).

As shown by the embodiments discussed above, the provision of an app orsimilar software product to obtain information regarding an injection,other medical treatment or the operation of other equipment may permitmore accurate and/or reliable recording of such information using adevice, such as a wearable electronic device, that is commonly availableand familiar to the user. Since the embodiments described above do notrequire the manufacture and distribution of a_specialized datacollection device, they may, potentially, reduce the costs andcomplexity of providing recordation and/or monitoring of treatment oroperations.

In the embodiments described above, video is captured, processed andanalyzed by one data collection device 20. However, in otherembodiments, the data collection device 20 may be configured to capturethe video using its camera 21 and then transmit the captured video to acomputer via a network, such as a wireless local area network, apersonal area network, a cellular communication network or the Internet,where the computer includes a processing arrangement configured toprocess and analyze the video to determine a dosage displayed in thedosage window 13.

While the embodiments above have been described in relation tocollecting data from an insulin injector pen, it is noted thatembodiments may be used for other purposes, such as monitoring ofinjections of other medicaments or other medical processes. Embodimentsmay also be used to for non-medical purposes, for example, in monitoringthe operation of other types of equipment for safety reasons.

1. A method of recording a medicament dose using a data collectiondevice, comprising: capturing, by a video camera of the data collectiondevice, a video showing a medicament dose indicator of an injector pen;determining a position of at least one of one or more characters in animage in said video; identifying the at least one character usingoptical character recognition software; and determining a medicamentdose indicated by the medicament dose indicator based on a result ofsaid identifying using said optical character recognition software,wherein the injector pen comprises a movable component for selectingsaid medicament dose to be dispensed.
 2. A method according to claim 1,wherein said data collection device is a wearable electronic device. 3.A method according to claim 1, comprising determining whether more thanone delivery of medicament is recorded in said video.
 4. A methodaccording to claim 3, comprising, in response to a determination thatmore than one delivery of medicament is recorded in said video,determining at least a second medicament dose based on at least onecharacter identified using the optical character recognition software onat least a second image of said medicament dose indicator in said videoand determining an overall medicament dose received by a user based onsaid determined medicament dose and said determined second medicamentdose.
 5. A method according to claim 3, comprising, in response to adetermination that more than one delivery of medicament is recorded insaid video, determining whether said more than one delivery ofmedicament includes one or more prime shots.
 6. A method according toclaim 1, comprising obtaining color information from said image andidentifying a type of medicament to be dispensed based on said colorinformation.
 7. A computer program comprising computer-readableinstructions that, when executed by one or more processors, cause one ormore operations to be performed, the one or more operations comprising:capturing, by a video camera of a data collection device, a videoshowing a medicament dose indicator of an injector pen; determining aposition of at least one of one or more characters in an image in saidvideo; identifying the at least one character using optical characterrecognition software; and determining a medicament dose indicated by themedicament dose indicator based on a result of said identifying usingsaid optical character recognition software, wherein the injector pencomprises a movable component for selecting said medicament dose to bedispensed.
 8. An application for a wearable electronic device, whereinsaid application comprises the computer program according to claim
 7. 9.A data collection apparatus for recording a medicament dose, the datacollection apparatus comprising: a video camera; and a processingarrangement configured to: capture a video using said video camera, saidvideo including at least one image of a medicament dose indicator of aninjector pen; determine a position of at least one of one or morecharacters in the at least one image; identify the at least onecharacter using optical character recognition software; and determine amedicament dose indicated by said medicament dose indicator based on aresult of said identifying using said optical character recognitionsoftware, wherein the injector pen comprises a movable component forselecting said medicament dose to be dispensed.
 10. A data collectionapparatus according to claim 9, comprising a wearable electronic deviceincluding said video camera.
 11. A data collection apparatus accordingto claim 10, wherein said wearable electronic device is configured to beworn on the head of a user.
 12. A data collection apparatus according toclaim 9, wherein said processing arrangement is configured to determinewhether more than one delivery of medicament is recorded in said video.13. A data collection apparatus according to claim 12, wherein theprocessing arrangement is configured to respond to a determination thatmore than one delivery of medicament is recorded in said video bydetermining at least a second medicament dose based on at least onecharacter identified using the optical character recognition software onat least a second image of said medicament dose indicator in said videoand determining an overall medicament dose received by a user based onsaid determined medicament dose and said determined second medicamentdose.
 14. A data collection apparatus according to claim 12, whereinsaid processing arrangement is configured to respond to a determinationthat more than one delivery of medicament is recorded in said video bydetermining whether said more than one delivery of medicament includesone or more prime shots.
 15. A medicament delivery system comprising: amedicament delivery device; and the data collection apparatus accordingto claim 9.